The Dutch Model of Data Collection Development for Official Surveys*

This article presents the methodological toolkit for data collection (particularly questionnaire) design and development in use at Statistics Netherlands (NCBS). The (re-)design of a continuous survey of living conditions is used to illustrate five development steps: (1) project preparation and risk analysis; (2) a qualitative study using exploratory tools in a questionnaire laboratory; (3) a qualitative study in the field; (4) a quantitative pilot in the field; and (5) implementation. A /inrtesting program determines how, in Steps 1—4, respondents are consulted about design issues concerning both the questionnaire and its administration. 'Respondent-friendly' design changes can thus be implemented before Step 5. The model is meant as a flexible framework for deciding when to test what and by which tool. This includes the possibility of steps being omitted, repeated, combined, or interchanged. Various test tools, like focus groups and cognitive interviews, are conceived as special cases of meta-interviews, by which (meta-) data are collected about and in addition to the intended data collection. The authors propose to use an eclectic classification of measurement error risks as a framework for describing risk hypotheses and test results. Risk hypodieses mainly serve to guarantee efficient use of means and time available for Steps 1-4. Unanticipated meta-information is obtained in the process. * The views and opinions expressed in this article are those of the authors and do not necessarily reflect official NCBS policies © World Association for Puilic Opinion Research tggy DUTCH MODEL OF DATA COLLECTION DEVELOPMENT 127 PRETESTING PROGRAMS AND A 'MODEL OF DATA COLLECTION DEVELOPMENT' PRETESTING FOR MEETING CERTAIN SURVEY OUTPUT QUALITY CRITERIA The quality of survey outcomes can be judged by criteria such as the following: (1) relevance (the target variable distributions are 'useful'); (2) timeliness (one keeps up with developments in the domain of study); (3) completeness (unit and item response, coverage); (4) validity (the survey instrument measures what it is intended to measure); (5) answer reliability (precision of individual measurements, interviewer bias); and (6) all kinds of non-observation criteria that, along with criteria 3-5, affect estimation or prediction quality (like sampling errors, processing errors, etc.). The non-observation errors involved in the latter category may be controlled to a large extent by sample and estimation design and by efficient and effective organization of the survey process. However, to satisfactorily meet criteria 1-5, respondent commitment, interviewer commitment, and user commitment are important factors to be taken into account. To investigate these factors systematically, one should perform a so-called pretesting program, which we define to be a series of interdependent empirical research steps for data collection design and development, meant to strike a balance between information demand and supply. As for respondent commitment, pretesting aims to optimize respondent friendliness by checking the quality of the fit between target variable operationalizations and respondentrelated data concepts (fit of content), as well as between questionnaire administration procedures and respondent data handling (fit of procedure). As for interviewer commitment, pretesting not only aims to foster interviewer friendliness (the interviewer should feel comfortable with the questionnaire and the interviewing procedure), but it may also use the interviewer as a valuable proxy informant about respondent commitment. If a new survey instrument is designed and developed, there may be more opportunity for pretesting than in the case of an existing survey serving to maintain a long-standing tradition of time-series data. Of course, in the domain of official surveys for national statistics, the distinction between the two cases is often not very sharp. For a coherent series of surveys, the quality criteria mentioned earlier may be dominated by the requirement of continuity. Even though some of the criteria 1-6 may be unsatisfactorily met, the overall quality of the desired time series may appear acceptable as long as, or precisely because, survey performance is stable over time. Pretesting of new surveys occurs, for example, in so-called business program redesign projects. One such program, dealing with the Statistics Netherlands' (NCBS) continuous survey on living conditions (POLS: Permanent Onderzoek Leefsituatie), is used as the principal example in this article. 128 INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH COMPOSING PRETESTING PROGRAMS ACCORDING TO A MODEL OF DATA COLLECTION DEVELOPMENT At Statistics Netherlands, a questionnaire laboratory, more fully called Questionnaire Design Resource CenteT (QDRC), offers research facilities and methods for designing and performing pretests as part of survey design and development (Akkerboom 1996). In these pretests, the focus is on collecting respondent-related meta-information in addition to just the data needed for the survey in question. Generally, such meta-information is information on what respondents think, do and feel while answering the survey questions. (This definition does not depend on any cognitive or social encounter model of the question-and-answer process.) Correspondingly, many pretesting tools belong to the general category of a meta-interview, which we define as an interview to collect respondent-related meta-information (there being analogous tools for user and interviewer consultation). Thus, for example, a focus group is considered to be an n-i (many-to-one) open meta-interview, and a cognitive interview is defined as a special case of a 1-1 (one-to-one) meta-interview. The meta-information obtained by respondent consultation usually has a much broader scope than merely the cognitive aspects of the question-and-answer process. The QDRC uses a (tentative) Model of Data Collection Development as a guideline for composing pretesting programs. (One might call this a Model of Questionnaire Development, insofar as the questionnaire is the focus of data collection.) The model is presented in Table ia. The title of the present article should be understood in the restrictive sense that (a) the model is Dutch because NCBS is the institute responsible for Dutch official statistics, and (b) it is the current best model the QDRC has to explain pretesting to its clients. Akkerboom and Luiten (1996) discussed a comprehensive example of selecting pretesting tools according to the model in Table ia. In this example the QDRC and its client were the sole actors responsible for the various survey development steps. However, this is not typical for QDRC contributions: pretesting programs are generally based on the same methodology, but the QDRC is usually only one out of many partners in survey development because of its large scale and scope. The main aim of the model is to provide a framework for deciding which design issues should be addressed when and by which tool. In case of designing a new survey instrument, all of the following steps can, in principle, be discerned: (1) definition and feasibility studies, (2) qualitative content test, (3) qualitative operational test, (4) quantitative test, and (5) implementation (with or without a 'test survey' or grand rehearsal). These steps are represented by the rows of Table ia. For each step, the model lists pretesting topics, tools and test size range (topics and tools are listed per step, whereas test size is specified per tool, given that such a tool would be selected in the relevant step). DUTCH MODEL OF DATA COLLECTION DEVELOPMENT 129 TABLE ia A model of data collection development: steps, topics, tools and test size